Student Ratings of the Importance of Survey Items, Multiplicative Factor Analysis, and the Validity of the Community of Inquiry Survey

This research builds upon prior validation studies of the Community of Inquiry (CoI) survey by utilizing multiple ratings measures to validate the survey’s tripartite structure (teaching presence, social presence, & cognitive presence). In prior studies exploring the construct validity of these 3 subscales, only respondents’ course ratings were utilized. This study asked participants to additionally rate the importance of each CoI survey item. Descriptive analyses of the gaps between course rating scores and the respective item importance ratings revealed that social presence items, perceived as the least important of the CoI subscales, yielded the gap scores with least variability, while gaps in teaching presence items revealed areas where instructors might focus more attention. Multiplicative scores for each item were computed as the product of an item’s course rating score and its corresponding importance rating. Even when including this additional measure of perceived importance, factor analysis of multiplicative scores (item rating * importance rating) supported the CoI model’s tripartite structure, and so prior validation studies. Introduction The issues addressed in this research are the relevance of the items in the Community of Inquiry (CoI) survey and their fidelity to the tripartite structure of the CoI model when viewed through the joint lenses of instantiation and importance. Previous validation studies of the CoI instrument have depended solely on students’ ratings of the degree to which teaching, social and cognitive presence were manifest in online courses; none have explored students’ perceptions of the importance of such manifestation. Whereas course ratings inform researchers of students’ perceptions of the degree to which a particular expectation is met in the online course, importance ratings enhance the perspectives students can offer by eliciting their input regarding the relevance of those expectations. The study reported in this paper explored the validity of the CoI instrument, and by extension the CoI framework itself, by incorporating both student ratings of the degree to which teaching, social, and cognitive presence were manifest in their courses and their ratings of the importance of teaching, social and cognitive presence as described in the CoI survey’s items. Background With almost four million students enrolled in online courses in the United States alone, and growth in online programs considered a priority at over 80% of institutions with significant online offerings (Allen & Seaman, 2008), understanding what constitutes high quality online teaching and learning is of the essence if we would preserve the high quality of American higher education. It is important that online administrators, designers and faculty understand not only that the processes of online learning differ significantly from their face-to-face counterparts, but that they understand how they differ and what pedagogical models best explain them (Bennett & Lockyer, 2004; Conrad, 2004). Developed by Garrison, Anderson, and Archer (2001), the Community of Inquiry (CoI) framework is a theoretical model that explains the processes of teaching and learning in online environments in terms of interactions among three overlapping presences: teaching, social and cognitive. Each of the presences are, in turn, conceptualized as consisting of multiple elements which are operationalized as observable indicators. For example, teaching presence is defined as the design, facilitation and direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes (Anderson, Rourke, Garrison & Archer, 2001), and it is viewed as arising out of effective practices in course design and organization, the facilitation of learning, and direct instruction. Social presence, the degree to which participants in computer-mediated communication feel affectively connected one to another, is seen as realized through affective expression, open communication, and group cohesion (Swan, Garrison & Richardson, 2009). Cognitive presence is defined as the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse in an online community of inquiry and conceptualized in terms of the four phases of the Practical Inquiry Model: triggering, exploration, integration, and resolution of ideas and concepts (Garrison, Anderson & Archer, 2001). Since its inception, the CoI model has been the most frequently cited theoretical model used to explain online educational experiences, with extensive research having been undertaken around both each of the individual presences (Arbaugh, 2007; Garrison & Arbaugh, 2007) and the CoI framework as a whole (Arbaugh et al, 2008). Many of these studies also address, either explicitly or implicitly, the validity of the CoI framework and/or its conceptualizations of the individual presences. Early CoI research included a good number of content analyses focused on the development of particular presences. In these studies, validation of the model consisted in identifying the hypothesized elements of each presence in online discussion transcripts. For example, Anderson et al. (2001) found evidence of all three elements of teaching presence, but interesting differences in the ways and extent to which they were employed, in the asynchronous discussions of two different online graduate courses. Swan (2003) similarly identified indicators of all three elements of social presence in the discussion transcripts of an online graduate course in education and documented the ways in which their uses and importance seemed to change over the life of the class. Content analyses concerned with identifying all four elements of cognitive presence have been more problematic, with several studies reporting that online discussions rarely moved beyond the exploration phase (Garrison, Anderson & Archer, 2001; Kanuka & Anderson, 1998; Luebeck & Bice, 2005). While various explanations for the low level of integration and resolution in online discussions have been suggested (including at least two in this special issue), researchers have found that discussions in which students have been challenged to resolve a problem and/or explicit facilitation has been provided do progress through to resolution (Meyer, 2003; Murphy, 2004; Shea & Bidjermo, 2009). Such findings also support the validity of the cognitive presence construct. Another strand of CoI investigations, which are rooted in social presence studies that actually predate the formulation of the CoI framework, involves survey research concerned initially with student perceptions of social (Richardson & Swan, 2003) and then teaching (Shea, Li, Swan & Pickett, 2005; Shea, Li & Pickett, 2006) presence. At first validation of the constructs of social and teaching presence in such survey research was implicit; if students perceived them (as indicated by their agreement with survey items), they must have at least some construct validity. Before long, however, researchers were attempting to statistically validate the the hypothesized three elements of teaching presence in particular through various analyses of survey responses. For example, Arbaugh and Hwang (2006) used structural equation modeling to confirm the presence of the three teaching presence categories in the survey responses of 191 MBA students. On the other hand, Shea, Li and Pickett (2006) conducted a large scale factor analysis the same year which suggested a two, not three, factor organization of teaching presence, made up of “design” and “directed facilitation” (an amalgamation of the facilitation and direct instruction categories) elements. Of course, validation of the elements of individual presences ignores a central facet of the CoI framework; namely that the model situates learning processes in the interaction of all three presences. Hence, CoI researchers have lately pushed to develop, and validate, a comprehensive CoI survey instrument which measures the elements of all three presences. The first such survey was developed by Arbaugh (2007) who applied factor analysis to the responses of 667 MBA students. His results revealed a four factor solution reminiscent of Shea, Li and Pickett’s (2006) findings concerning teaching presence considered by itself; the factor analysis cleanly distinguished between social and cognitive presence, but split teaching presence into the same two factors identified by Shea et al – design and directed facilitation. However, the survey Arbaugh used consisted of 18 teaching presence, 8 social presence, and only 4 cognitive presence items making it difficult to tell whether the bifurcation of teaching presence was the result of an over abundance of items or a real difference in student perceptions. Meanwhile, Arbaugh and others were working to devise a common survey that major CoI researchers could agree on. Commonalities between items in previous instruments were reconciled and, where appropriate, new items created to fully capture each of the presences (Arbaugh, Cleveland-Innes, Diaz, Garrison, Ice, Richardson, Shea & Swan, 2008). The resultant 34 item instrument was administered to 287 students taking online courses at four institutions in the US and Canada in the summer of 2007. Factor analysis of their responses to the Likert scale items, using principal component analysis with obliminal rotation, not only identified the three factors predicted by CoI model, but did not support a framework involving more than the anticipated three (Swan, Richardson, Ice, Garrison, Cleveland-Innes, & Arbaugh, 2008). Subsequent confirmatory factor analyses of responses to the new CoI survey, including a large scale study involving 2,159 online learners (Shea & Bidjerano, 2009) and the Garrison, Cleveland-Innes and Fung research reported in this issue, have likewise supported the three factor solution, and so the construct validity of the CoI framework. Although the various studies and methodologies described above together strongly confirm the validity of the CoI framework, they do not tell us anything about how important students think the processes it describes are. Indeed, that is okay. Many conventional factor analyses conducted for the purposes of construct validation of surveys used in educational evaluation rely solely on respondent’s perceptions of the existence of a particular phenomenon. The methodology for the study reported in this paper, however, is deliberately more expansive in seeking respondents’ input, asking them to offer value judgments as well about the importance of the individual CoI survey items. If investigators trust students in online courses to recognize the existence of these measurable characteristics, they should likewise have some confidence in their ability judge the importance of those characteristics. Although students’ item-importance ratings are not meant to be the definitive measure of the importance of individual CoI survey items, their perspective is nevertheless valuable and crucial to the validation of the instrument. This methodological approach is consistent with Paulo Freire’s teacher and learner-centered philosophy of education. Freire (1958) recommended a dialogical approach in which investigators and the people who would normally be considered objects of investigation should act together as co-investigators. Allowing students enrolled in online courses the opportunity to offer their perspectives on what is important helps achieve Freire’s suggested aim and better informs research intended to improve teaching and learning in online environments. Eliot Eisner’s perspectives on educational evaluation also support the value in seeking student’s perceptions of what is important in online pedagogy. In arguing for the importance of educational connoisseurship in the creation of educational criticism, Eisner (1979) distinguishes between recognition, which he views as an act of classification, and perception, iwhich he sees as a form of exploration.. This study recognizes the importance of both: Students’ recognition of the existence of a particular characteristic and their perception of its respective importance both provide valuable information which can ultimately be used to enhance online teaching and learning. Pragmatic examples add support to these philosophical and foundational arguments for the utility of feedback from online students regarding the importance of CoI survey items. For example, multiplicative scores employing similar dual ratings commonly are used in Quality of Life surveys (Hsieh, 2004). Although Trauer and Mackinnon (2001) suggest that the ratings of satisfaction and importance employed in such studies are redundant, importance ratings are not redundant with the identification of the degree to which a particular CoI indicator is met. This study, therefore, explored the validity of the CoI instrument and by extension the CoI framework itself by incorporating both student ratings of the degree to which teaching, social, and cognitive presence were manifest in their courses and their ratings of the importance of teaching, social and cognitive presence as described in the CoI survey’s items. The following research questions guided the study:  RQ1: Does factor analysis confirm the construct validity of the CoI instrument (and model) when multiplicative scores that consider item importance as well as the degree to which items are met (item rating * importance rating) are employed?  RQ2: How do descriptive gap analyses comparing mean course ratings and mean item importance ratings inform the construct validity of the CoI framework? Methods and Results This study builds upon research utilizing the Community of Inquiry (CoI) Framework survey by expanding the construct validation of the instrument. Whereas previous studies (Arbaugh, et al., 2008; Swan et al., 2008; Shea & Bidjerano, 2009) utilized a singular rating dimension (i.e. survey items were used to rate the course), this study adds a second dimension by asking respondents to also rate the relative importance of each item in the CoI survey. The resulting enhanced CoI survey is given in Appendix A. As opposed to simply comparing factor analyses results for the two sets of ratings, this study reports a single factor analysis on multiplicative scores computed as the product of the course ratings and item-importance ratings. Course ratings utilized a conventional Likert scale (Strongly Disagree=1; Strongly Agree=5), while item-importance ratings utilized an ordinal scale with the same range of quantitative values (1=Unimportant; 2=Somewhat Important; 3=Important; 4=Very Important; 5=Extremely Important). The multiplicative scores, therefore, range from 1 to 25 in possible value. In those instances where a student Strongly Disagrees that a particular characteristic exists that she also perceives to be Unimportant, the respective multiplicative score equals 1 (1 x 1 = 1). At the other extreme, when a student Strongly Agrees that an Extremely Important element is manifest in her online course, the respective multiplicative score equals 25 (5 x 5 = 25). Scores that fall somewhere in the middle of this possible range indicate a variety of possible combinations for course rating and item importance scores. The 34 item enhanced CoI survey instrument was administered to an approximately even mix of graduate and undergraduate students at four US colleges and universities.. A total of n=413 students volunteered to complete the survey, yielding an average response rate of about 40%. . A single respondent failed to provide complete data, yielding a final sample of n=412. In the sections which follow, analyses of the survey data are described and their findings summarized. Results are presented in three sections. In the first sections, results are provided for students’ ratings of their courses using the CoI items. In the second section, results are provided for students’ ratings of the relative importance of each of the CoI items as they relate to online learning. The third section compares results for course and item-importance ratings, and the factor anslysis of multiplicative scores (course rating * item importance scors) is discussed in the fourth section. Student Ratings of Conventional CoI Items Students’ ordinal responses to items used to assess the degree to which teaching, social, and cognitive presence were manifest in their courses were scored using a five point Likert-type scale (1=Strongly Disagree to 5=Strongly Agree). Mean responses for the 34 items ranged from 4.44 for Item #4 (The instructor clearly communicated important due dates / time frames for learning activities) to 3.66 for Item #16 (Online or web-based communication is an excellent medium for social interaction). Standard deviations were highest for Item #12 (s.d. = 1.11) (The instructor provided feedback that helped me understand my strengths and weaknesses relative to the course's goals and objectives), and lowest for Item #2 (s.d. = 0.80) (The instructor clearly communicated important course goals). When considering all respondents’ ratings for the course, teaching presence items collectively yielded a mean score of 4.22 (s.d. = 0.93). social presence items collectively yield a mean score of 3.98 (s.d. = 0.99), and cognitive presence items yield a mean score of 4.08 (s.d. = 0.89). Cronbach’s Alpha revealed internal consistencies equal to 0.96 for teaching presence items, 0.92 for social presence items, and 0.95 for cognitive presence items. Student Ratings of the Importance of CoI Survey Items Students’ ratings of the importance of CoI items were scored using an ordinl scale (1=Unimportant; 2=Somewhat Important; 3=Important; 4=Very Important; 5=Extremely Important). This ordinal scale was used because it provides multiple nuances of important from which students can choose in anticipation that respondents would be reluctant to rate items as unimportant. Mean responses for the 34 items ranged from 4.52 for Item #4 (The instructor clearly communicated important due dates / time frames for learning activities) to 2.84 for Item #15 (I was able to form distinct impressions of some course participants). Standard deviations were highest for Item #16 (s.d. = 1.26) (Online or web-based communication is an excellent medium for social interaction), and lowest for Item #4 (s.d. = 0.70) (The instructor clearly communicated important due dates / time frames for learning activities). When considering all respondents’ ratings for the importance of CoI items, teaching presence items collectively yield a mean score of 4.05 (s.d. = 0.95). social presence items collectively yield a mean score of 3.52 (s.d. = 1.18), and cognitive presence items yield a mean score of 3.77 (s.d. = 0.99). Cronbach’s Alpha revealed internal consistencies equal to 0.91 for the importance of teaching presence items, 0.94 for importance of social presence items, and 0.94 for importance of cognitive presence items. Descriptive Comparisons of Course and Importance Ratings of CoI Items Although the ordinal scales used in this study are not synonymous (Strongly Agree to Strongly Disagree vs. Unimportant to Extremely Important), comparing their respective quantitative values (1-5) for each item and/or presence subscale is useful for identifying discrepancies. Of particular interest are those items which receive a relatively low presence rating but a high importance rating, as such results highlight areas which students perceive as especially important but for which their expectations are not well met.. The following ladder graphs help illustrate how such discrepancies are used in what is often referred to as Gap Analysis. Mean ratings are shown along the y axis, and the “rungs” of the ladder connect the mean value for the course rating on that item with the mean rating for its importance. The first ladder graph (Figure 1) shown below illustrates overall means for the three presence subscales (these values are rescaled 1 (low) to 5 (high) for ease of interpretation). Figure 1. Comparison of Course and Importance Ratings by Presence As this ladder graph demonstrates, ratings for the importance of the CoI items were often quantitatively less than the ratings for the course. Even though these two ratings are not synonymous, the trend reveals that while course ratings for the three presence subscales are all close to 4.0 (teaching presence = 4.20; social presence = 3.97; cognitive presence = 4.08), their respective importance ratings are much more dispersed (importance of teaching presence = 4.09; importance of social presence = 3.34; importance of cognitive presence = 3.77). This may be explained in part because the ordinal scale for importance utilizes 4 of the 5 anchors to distinguish among levels of importance, whereas a Likert scale splits anchor points evenly among levels of agreement and disagreement. However, as illustrated by the diverging slopes in the ladder graph shown in Figure 1, students believed teaching presence to be more important than both cognitive presence and social presence, and they valued social presence least of all. When using ladder graphs to illustrate gap analysie of ratings on individual items, different patterns emerge among items designed to measure elements of each of the three presences. The ladder graph below (Figure 2) examines the respective means for course and importance ratings on each of the teaching presence items. Four teaching presence items yielded importance rating scores higher than the corresponding ratings of presence manifest. Indeed, this ladder graph includes instances of all possible relationships between course and importance ratings; namely examples in which: 1) course ratings for a particular item were higher than importance ratings, 2) importance ratings for a particular item were higher than course ratings, and; 3) the two ratings for a particular item were about the same. Figure 2. Comparison of Course and Importance Ratings on Teaching Presence Items One way to interpret ladder graphs is to compare and contrast instances in which importance ratings are much higher than course ratings, and vice versa. For example in Figure 2, Item #10 and Item #13 both received similar course ratings (means equal to 4.06 and 4.09, respectively), but the importance rating for Item #10 was much lower in scale (mean=3.56), and the importance rating for Item#13 was much higher (mean=4.40). The findings thus reveal that, although students perceived the degree to which these two teaching presence indicators were manifest in their courses as about the same, they felt that Item #13 (The instructor provided feedback in a timely fashion) represented a much more important element than did Item #10 (Instructor actions reinforced the development of a sense of community among course participants). Findings of this sort could be extremely useful to course instructors trying to apportion their limited teaching presence resources to greatest effect. For example, the contrast explored above might indicate that a higher priority be placed on providing timely feedback. Another strategy for exploring the information represented in ladder graphs is to compare patterns between graphs. For example, a more parallel structure of course to importance rating can be seen in the ladder graph depicting social presence items from the CoI survey (Figure 3). Figure 3 shows that social presence items received overall lower mean ratings on importance than for manifest presence, and that the respective differences between the two ratings for each item were similar. Figure 3. Comparison of Course and Importance Ratings on Social Presence Items When examining a ladder graph depicting the relationship between course and importance ratings on CoI cognitive presence items (Figure 4), the patterns seem to fall somewhere between those depicted in the ladder graphs for teaching presence and social presence. On the majority of items, importance ratings were consistently lower than the respective course rating. However, for item #33 (I have developed solutions to course problems that can be applied in practice.), the two ratings are about the same, indicating the high value students placed on this item. Figure 4. Comparison of Course and Importance Ratings on Cognitive Presence Items Factor Analysis of Multiplicative Scores The sample size (n=412) for this study is reasonably adequate for a corresponding factor analysis according to many rules of thumb offered. The study meets Kass & Tinley’s (1979) recommendation for 5 to 10 participants per item, as well as Nunnally’s (1978) more stringent standards requiring at least 10. Some authors suggest absolute sample sizes of n=300 being adequate (Tabachnik & Fidell, 2007). Comrey & Lee (1992) rate sample sizes of 200 as Fair and 300 as Good, while Tabachnik and Fidell (2007) rate absolute sample sizes of 300 are Adequate. The Principal Components approach in SPSS version 17.0 was used to explore the subscale structure of the 34 multiplicative items related to the CoI inventory. Assuming some degree of association among importance ratings for items addressing teaching, social, and cognitive presence (Heckman & Annabi, 2005; Garrison et al., 2004), oblique rotation (Direct Obliminal in SPSS) was utilized with the default value δ=0 specified to reasonably limit the level of correlation among the factors. The use of an oblique rotation was justified on the theoretical grounds that the three presences are considered to be interdependent, thus their multiplicative scores would be considered independent as well. The Keyser-Meyer-Olkin (KMO) measure of sampling adequacy was 0.95, suggesting factor analysis should yield distinct and reliable factors given the data utilized. Table 1 and Figure 5 show the eigenvalues and the scree plot for the principal components analysis of multiplicative scores for the CoI survey items. When specifying a three factor solution within SPSS, factor loadings for the 34 multiplicative items support the validity of the Coi framework as teaching, social, and cognitive presence load cleanly as expected. One caveat is that for item #22 (Online discussions help me to develop a sense of collaboration.), the second highest factor loading differed from the highest factor loading by only 0.004 in absolute value, suggesting the item does not distinguish markedly between the teaching and social presence factors. The three factors together accounted for 61.9% of the total variance in CoI item scores. Table 1. Eigenvalues for Principal Component Analysis with a Three Factor Solution Specified Figure 5. Scree Plot for Principal Component Analysis with a Three Factor Solution Specified Table 2. Factor Loadings for Multiplicative Item Scores when Selecting a Three-Factor Solution